
 <!DOCTYPE HTML>
<html>
<head><meta name="generator" content="Hexo 3.9.0">
  <meta charset="UTF-8">
  
    <title>ELK的一些技巧和知识记录 | Zong&#39;s blog</title>
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=3, minimum-scale=1">
    
    <meta name="author" content="Zong">
    
    <meta name="description" content="总体架构最近自己搭建了一套ELK，在摸索中总结了一下，这个post不是一个step by step安装教程。现在初步的架构是这样：


logs表示从客户端收集到的日志，收集日志通常使用logstash、flume、自己写的python工具等。kafka是作为一个缓冲和标准的数据传输接口。logst">
    
    
    
    
    
    <link rel="icon" href="/img/favicon.ico">
    
    
    <link rel="apple-touch-icon" href="/img/pacman.jpg">
    <link rel="apple-touch-icon-precomposed" href="/img/pacman.jpg">
    
    <link rel="stylesheet" href="/css/style.css">
</head>
</html>
  <body>
    <header>
      <div>
		
			<div id="imglogo">
				<a href="/"><img src="/img/logo.svg" alt="Zong&#39;s blog" title="Zong&#39;s blog"/></a>
			</div>
			
			<div id="textlogo">
				<h1 class="site-name"><a href="/" title="Zong&#39;s blog">Zong&#39;s blog</a></h1>
				<h2 class="blog-motto">日常积累，技术分享</h2>
			</div>
			<div class="navbar"><a class="navbutton navmobile" href="#" title="Menu">
			</a></div>
			<nav class="animated">
				<ul>
					<ul>
					 
						<li><a href="/">Home</a></li>
					
						<li><a href="/archives">Archives</a></li>
					
						<li><a href="/categories/运维">运维</a></li>
					
						<li><a href="/categories/容器架构">容器架构</a></li>
					
					<li>
					
					<form class="search" action="//baidu.com/s" method="get" accept-charset="utf-8">
						<label>Search</label>
						<input type="text" id="search" name="wd" autocomplete="off" maxlength="20" placeholder="Search" />
                        <input name=tn type=hidden value="bds">
                        <input name=cl type=hidden value="3">
                        <input name=ct type=hidden value="2097152">
						<input type="hidden" name="si" value="www.lstop.pub">
					</form>
					
					</li>
				</ul>
			</nav>			
</div>

    </header>
    <div id="container">
      <div id="main" class="post" itemscope itemprop="blogPost">
	<article itemprop="articleBody"> 
		<header class="article-info clearfix">
  <h1 itemprop="name">
    
      <a href="/2017/06/19/ELK的一些技巧和知识记录/" title="ELK的一些技巧和知识记录" itemprop="url">ELK的一些技巧和知识记录</a>
  </h1>
  <p class="article-author">By
    
      <a href="http://www.lstop.pub" title="Zong">Zong</a>
    </p>
  <p class="article-time">
    <time datetime="2017-06-19T08:13:51.000Z" itemprop="datePublished">2017-06-19</time>
    
  </p>
</header>

	<div class="article-content">
		
		
		<div id="toc" class="toc-article">
			<strong class="toc-title">Contents</strong>
		<ol class="toc"><li class="toc-item toc-level-1"><a class="toc-link" href="#总体架构"><span class="toc-number">1.</span> <span class="toc-text">总体架构</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#elasticsearch"><span class="toc-number">2.</span> <span class="toc-text">elasticsearch</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#kafka"><span class="toc-number">3.</span> <span class="toc-text">kafka</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#logstash、flume、kafka-connect-elasticsearch"><span class="toc-number">4.</span> <span class="toc-text">logstash、flume、kafka-connect-elasticsearch</span></a></li></ol>
		</div>
		
		<h1 id="总体架构"><a href="#总体架构" class="headerlink" title="总体架构"></a>总体架构</h1><p>最近自己搭建了一套ELK，在摸索中总结了一下，这个post不是一个step by step安装教程。<br>现在初步的架构是这样：</p>
<div id="flowchart-0" class="flow-chart"></div>

<p>logs表示从客户端收集到的日志，收集日志通常使用logstash、flume、自己写的python工具等。<br>kafka是作为一个缓冲和标准的数据传输接口。<br>logstash作为kafka和es的桥梁、原本想使用其他方式、比如flume、kafka-connect-elasticsearch等工具直接把json格式录入es、不过遇到版本问题，我安装的es版本5.4太新了，一些工具都没update，就先用logstash吧<br>当前算是初代架构，后面看情况调整。</p>
<h1 id="elasticsearch"><a href="#elasticsearch" class="headerlink" title="elasticsearch"></a>elasticsearch</h1><p>elasticsearch是一个全文索引的存储引擎吧，能快速搜索海量数据<br>安装还是使用apt</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"># 导入PGP Key</span><br><span class="line">wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -</span><br><span class="line"># 从APT库安装</span><br><span class="line">sudo apt-get install apt-transport-https</span><br><span class="line">echo &quot;deb https://artifacts.elastic.co/packages/5.x/apt stable main&quot; | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list</span><br><span class="line">sudo apt-get update</span><br><span class="line">sudo apt-get install elasticsearch</span><br></pre></td></tr></table></figure>

<blockquote>
<p>ps 如果安装es6的话用 </p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">&gt; echo &quot;deb https://artifacts.elastic.co/packages/6.x/apt stable main&quot; | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list</span><br><span class="line">&gt;</span><br></pre></td></tr></table></figure>
</blockquote>
<p>基本上导入了上面的elastic源就能apt安装logstash、elasticsearch、kibana，很方便，而且版本都最新。</p>
<p>配置就不多写了，网上也很多，下面记录一下常用的管理命令</p>
<ol>
<li><p>查看es的index</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">curl -XGET &apos;wx_es_1:9200/_cat/indices?v&apos;</span><br><span class="line">health status index                             uuid                   pri rep docs.count docs.deleted store.size pri.store.size</span><br><span class="line">yellow open   logstash-wgw-accesslog-2017.06.07 3lj8lgN5T9qYU1BW2y07hQ   5   1       2548            0      4.6mb          4.6mb</span><br><span class="line">yellow open   logstash-wgw-accesslog-2017.06.06 GC-ElhrLQu2rNfYtvR4XSg   5   1    2335781            0      2.1gb          2.1gb</span><br><span class="line">yellow open   packetbeat-2017.06.08             KsS2uikmTES3OkZwPu5TpQ   5   1    2340461            0    487.6mb        487.6mb</span><br></pre></td></tr></table></figure>
</li>
<li><p>删除一个index</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">curl -XDELETE &apos;wx_es_1:9200/wx-qiniu/?pretty&apos;</span><br></pre></td></tr></table></figure>
</li>
<li><p>添加模板<br>比如下面template_2模板就定义了所有wx-*索引的logtime字段格式，类似 15/Jun/2017:15:36:10 +0800</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line">curl -XPUT http://wx_es_1:9200/_template/template_2 -d &apos;&#123;</span><br><span class="line">    &quot;template&quot; : &quot;wx-*&quot;,</span><br><span class="line">    &quot;mappings&quot; : &#123;</span><br><span class="line">        &quot;type1&quot; : &#123;</span><br><span class="line">            &quot;properties&quot;: &#123;</span><br><span class="line">			    &quot;logtime&quot;: &#123;</span><br><span class="line">				&quot;type&quot; : &quot;date&quot;,</span><br><span class="line">				&quot;format&quot; : &quot;dd/MMM/YYYY:HH:mm:ss Z&quot;</span><br><span class="line">				&#125;</span><br><span class="line">			&#125;</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;&apos;</span><br></pre></td></tr></table></figure>

</li>
</ol>
<p>更详细参考<br><a href="http://www.cnblogs.com/didda/p/4758061.html" target="_blank" rel="noopener">ElasticSearch 模板文件配置</a><br><a href="https://kibana.logstash.es/content/elasticsearch/template.html" target="_blank" rel="noopener">ELKstack中文指南</a></p>
<h1 id="kafka"><a href="#kafka" class="headerlink" title="kafka"></a>kafka</h1><p>kafka是作为一个缓冲和标准的数据传输接口，kafka处理消息的速度据说是rabbitmq的N倍，第一次尝试。<br>kafka使用了二进制包安装，解压就ok，启动需要依赖zookeeper<br>常用命令：</p>
<ol>
<li><p>查看kafka topic</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bin/kafka-topics.sh --list --zookeeper zk1-test:2182</span><br></pre></td></tr></table></figure>
</li>
<li><p>使用console consumer<br>调试的时候方便输出消息查看</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bin/kafka-console-consumer.sh  --topic wx-qiniu --bootstrap-server  wx_es_1:9092</span><br></pre></td></tr></table></figure>
</li>
<li><p>使用console producter<br>调试的时候方便输入消息数据</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bin/kafka-console-producer.sh --topic logs-wx --broker-list wx_es_1:9092</span><br></pre></td></tr></table></figure>
</li>
<li><p>删除topic<br>server.properties中的delete.topic.enable=true需要设置成true，才能使用命令直接删除topic和数据</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">bin/kafka-topics.sh --delete --topic testl --zookeeper zk1-test:2182</span><br></pre></td></tr></table></figure>

</li>
</ol>
<h1 id="logstash、flume、kafka-connect-elasticsearch"><a href="#logstash、flume、kafka-connect-elasticsearch" class="headerlink" title="logstash、flume、kafka-connect-elasticsearch"></a>logstash、flume、kafka-connect-elasticsearch</h1><p>这里说的其实是kafka与es的数据通信问题，logstash是elastic官方原生支持，更新快，但是感觉作为中间通信层，不够简洁高效（猜测的）。<br>flume作为常见的log收集和传输工具，结构上实现了source-&gt;channel-&gt;sink 的形式，输入输出比较灵活，例如可以多个sink分别把数据传入es和jstrom等。<br>kafka-connect-elasticsearch是最先考虑的，直接在log收集时把数据格式化成json，通过kafka的connect实现数据传入es，版本问题不支持。</p>
<p>长远来说，flume可能更优先考虑，也可能是多种方式，反正比较灵活。<script src="https://cdnjs.cloudflare.com/ajax/libs/raphael/2.2.7/raphael.min.js"></script><script src="https://cdnjs.cloudflare.com/ajax/libs/flowchart/1.6.5/flowchart.min.js"></script><textarea id="flowchart-0-code" style="display: none">op1=>operation: logs|past
op2=>operation: Kafka|past
op3=>operation: logstash|past
op4=>operation: elasticsearch|past
op5=>operation: kibana|past

op1(right)->op2(right)->op3(right)->op4(right)->op5</textarea><textarea id="flowchart-0-options" style="display: none">{"scale":1,"line-width":2,"line-length":50,"text-margin":10,"font-size":12}</textarea><script>  var code = document.getElementById("flowchart-0-code").value;  var options = JSON.parse(decodeURIComponent(document.getElementById("flowchart-0-options").value));  var diagram = flowchart.parse(code);  diagram.drawSVG("flowchart-0", options);</script></p>
  
	</div>
		<footer class="article-footer clearfix">

  <div class="article-tags">
  
  <span></span> <a href="/tags/elk/">elk</a>
  </div>


<div class="article-categories">
  <span></span>
  <a class="article-category-link" href="/categories/运维/">运维</a>
</div>



<div class="article-share" id="share">

  <div data-url="http://www.lstop.pub/2017/06/19/ELK的一些技巧和知识记录/" data-title="ELK的一些技巧和知识记录 | Zong&#39;s blog" data-tsina="" class="share clearfix">
  </div>

</div>
</footer>   	       
	</article>
	
<nav class="article-nav clearfix">
 
 <div class="prev" >
 <a href="/2017/10/16/docker构造hadoop-spark镜像/" title="docker构造hadoop+spark镜像">
  <strong>PREVIOUS:</strong><br/>
  <span>
  docker构造hadoop+spark镜像</span>
</a>
</div>


<div class="next">
<a href="/2017/05/26/一直以来对tcp-tw-recycle参数的误解及tcp-timestamps/"  title="一直以来对tcp_tw_recycle参数的误解及tcp_timestamps">
 <strong>NEXT:</strong><br/> 
 <span>一直以来对tcp_tw_recycle参数的误解及tcp_timestamps
</span>
</a>
</div>

</nav>

	
</div>  
      <div class="openaside"><a class="navbutton" href="#" title="Show Sidebar"></a></div>

  <div id="toc" class="toc-aside">
  <strong class="toc-title">Contents</strong>
  <ol class="toc"><li class="toc-item toc-level-1"><a class="toc-link" href="#总体架构"><span class="toc-number">1.</span> <span class="toc-text">总体架构</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#elasticsearch"><span class="toc-number">2.</span> <span class="toc-text">elasticsearch</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#kafka"><span class="toc-number">3.</span> <span class="toc-text">kafka</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#logstash、flume、kafka-connect-elasticsearch"><span class="toc-number">4.</span> <span class="toc-text">logstash、flume、kafka-connect-elasticsearch</span></a></li></ol>
  </div>

<div id="asidepart">
<div class="closeaside"><a class="closebutton" href="#" title="Hide Sidebar"></a></div>
<aside class="clearfix">

  
<div class="tagslist">
	<p class="asidetitle">Tags</p>
		<ul class="clearfix">
		
			<li><a href="/tags/Airtest/" title="Airtest">Airtest<sup>1</sup></a></li>
		
			<li><a href="/tags/DNS/" title="DNS">DNS<sup>1</sup></a></li>
		
			<li><a href="/tags/GitLab/" title="GitLab">GitLab<sup>1</sup></a></li>
		
			<li><a href="/tags/K8s/" title="K8s">K8s<sup>8</sup></a></li>
		
			<li><a href="/tags/Linux/" title="Linux">Linux<sup>1</sup></a></li>
		
			<li><a href="/tags/MongoDB/" title="MongoDB">MongoDB<sup>2</sup></a></li>
		
			<li><a href="/tags/OpenWrt/" title="OpenWrt">OpenWrt<sup>1</sup></a></li>
		
			<li><a href="/tags/Python/" title="Python">Python<sup>2</sup></a></li>
		
			<li><a href="/tags/RabbitMQ/" title="RabbitMQ">RabbitMQ<sup>1</sup></a></li>
		
			<li><a href="/tags/calico/" title="calico">calico<sup>1</sup></a></li>
		
			<li><a href="/tags/cdn/" title="cdn">cdn<sup>1</sup></a></li>
		
			<li><a href="/tags/docker/" title="docker">docker<sup>3</sup></a></li>
		
			<li><a href="/tags/docker-registry/" title="docker registry">docker registry<sup>1</sup></a></li>
		
			<li><a href="/tags/elasticsearch/" title="elasticsearch">elasticsearch<sup>3</sup></a></li>
		
			<li><a href="/tags/elk/" title="elk">elk<sup>3</sup></a></li>
		
			<li><a href="/tags/k8s/" title="k8s">k8s<sup>3</sup></a></li>
		
			<li><a href="/tags/kubernetes/" title="kubernetes">kubernetes<sup>1</sup></a></li>
		
			<li><a href="/tags/nginx/" title="nginx">nginx<sup>1</sup></a></li>
		
			<li><a href="/tags/python/" title="python">python<sup>1</sup></a></li>
		
			<li><a href="/tags/tomcat/" title="tomcat">tomcat<sup>1</sup></a></li>
		
		</ul>
</div>


  <div class="linkslist">
  <p class="asidetitle">Links</p>
    <ul>
      <li><a href="http://www.v2ex.com/?r=zong400" target="_blank" title="V2EX">V2EX</a></li>
      <li><a href="http://hexo.io" target="_blank" title="Hexo">Hexo</a></li>
	  <li><a href="https://promotion.aliyun.com/ntms/yunparter/invite.html?userCode=s0bh6uzq" target="_blank" title="阿里云">阿里云</a></li>
	  <li><a href="https://cloud.tencent.com/redirect.php?redirect=1014&cps_key=5bd9deb84d4d9f34b65fb934e12d03e3&from=console" target="_blank" title="腾讯云">腾讯云</a></li>
    </ul>
</div>


</aside>
</div>
    </div>
    <footer><div id="footer" >
	
	
	<div class="social-font" class="clearfix">
		
		
		
		
	</div>
		<p class="copyright">Hosted by <a href="https://pages.coding.me/" target="_blank" title="Coding Pages">Coding Pages</a></p>
		<p class="copyright">Powered by <a href="http://hexo.io" target="_blank" title="hexo">hexo</a> and Theme by <a href="https://github.com/wizicer/iceman" target="_blank" title="Iceman">Iceman</a> © 2020 
		
		<a href="http://www.lstop.pub" target="_blank" title="Zong">Zong</a>
		
		</p>
</div>
</footer>
    <script src="//cdn.staticfile.org/jquery/2.1.0/jquery.min.js"></script>
<script type="text/javascript">
$(document).ready(function(){ 
  $('.navbar').click(function(){
    $('header nav').toggleClass('shownav');
  });
  var myWidth = 0;
  function getSize(){
    if( typeof( window.innerWidth ) == 'number' ) {
      myWidth = window.innerWidth;
    } else if( document.documentElement && document.documentElement.clientWidth) {
      myWidth = document.documentElement.clientWidth;
    };
  };
  var m = $('#main'),
      a = $('#asidepart'),
      c = $('.closeaside'),
      o = $('.openaside');
  $(window).resize(function(){
    getSize(); 
    if (myWidth >= 1024) {
      $('header nav').removeClass('shownav');
    }else
    {
      m.removeClass('moveMain');
      a.css('display', 'block').removeClass('fadeOut');
      o.css('display', 'none');
      
      $('#toc.toc-aside').css('display', 'none');
        
    }
  });
  c.click(function(){
    a.addClass('fadeOut').css('display', 'none');
    o.css('display', 'block').addClass('fadeIn');
    m.addClass('moveMain');
  });
  o.click(function(){
    o.css('display', 'none').removeClass('beforeFadeIn');
    a.css('display', 'block').removeClass('fadeOut').addClass('fadeIn');      
    m.removeClass('moveMain');
  });
  $(window).scroll(function(){
    o.css("top",Math.max(80,260-$(this).scrollTop()));
  });
});
</script>

<script type="text/javascript">
$(document).ready(function(){ 
  var ai = $('.article-content>iframe'),
      ae = $('.article-content>embed'),
      t  = $('#toc'),
      h  = $('article h2')
      ah = $('article h2'),
      ta = $('#toc.toc-aside'),
      o  = $('.openaside'),
      c  = $('.closeaside');
  if(ai.length>0){
    ai.wrap('<div class="video-container" />');
  };
  if(ae.length>0){
   ae.wrap('<div class="video-container" />');
  };
  if(ah.length==0){
    t.css('display','none');
  }else{
    c.click(function(){
      ta.css('display', 'block').addClass('fadeIn');
    });
    o.click(function(){
      ta.css('display', 'none');
    });
    $(window).scroll(function(){
      ta.css("top",Math.max(140,320-$(this).scrollTop()));
    });
  };
});
</script>


<script type="text/javascript">
$(document).ready(function(){ 
  var $this = $('.share'),
      url = $this.attr('data-url'),
      encodedUrl = encodeURIComponent(url),
      title = $this.attr('data-title'),
      tsina = $this.attr('data-tsina');
  var html = [
  '<a href="#" class="overlay" id="qrcode"></a>',
  '<div class="qrcode clearfix"><span>扫描二维码分享到微信朋友圈</span><a class="qrclose" href="#share"></a><strong>Loading...Please wait</strong><img id="qrcode-pic" data-src="http://s.jiathis.com/qrcode.php?url=' + encodedUrl + '"/></div>',
  '<a href="#textlogo" class="article-back-to-top" title="Top"></a>',
  '<a href="https://www.facebook.com/sharer.php?u=' + encodedUrl + '" class="article-share-facebook" target="_blank" title="Facebook"></a>',
  '<a href="#qrcode" class="article-share-qrcode" title="QRcode"></a>',
  '<a href="https://twitter.com/intent/tweet?url=' + encodedUrl + '" class="article-share-twitter" target="_blank" title="Twitter"></a>',
  '<a href="http://service.weibo.com/share/share.php?title='+title+'&url='+encodedUrl +'&ralateUid='+ tsina +'&searchPic=true&style=number' +'" class="article-share-weibo" target="_blank" title="Weibo"></a>',
  '<span title="Share to"></span>'
  ].join('');
  $this.append(html);
  $('.article-share-qrcode').click(function(){
    var imgSrc = $('#qrcode-pic').attr('data-src');
    $('#qrcode-pic').attr('src', imgSrc);
    $('#qrcode-pic').load(function(){
        $('.qrcode strong').text(' ');
    });
  });
});     
</script>









  </body>
</html>

